Evaluation of the Performance of Drug-Drug Interaction Screening Software in Community and Hospital Pharmacies

BACKGROUND: Computerized drug-drug interaction (DDI) screening is widely used to identify potentially harmful drug combinations in the inpatient and outpatient setting. OBJECTIVES: To evaluate the performance of drug-drug interaction (DDI) screening software in identifying select clinically significant DDIs in pharmacy computer systems in community and hospital pharmacies. METHODS: Ten community pharmacies and 10 hospital pharmacies in the Tucson metropolitan area were invited to participate in the study in 2004. To test the performance of each of the systems used by the pharmacies, 25 medications were used to create 6 mock patient profiles containing 37 drug-drug pairs, 16 of which are clinically meaningful DDIs that pose a potential risk to patient safety. Each profile was entered into the computer pharmacy system, and the system response in terms of the presence or absence of a DDI alert was recorded for each drug pair. The percentage of correct responses and the sensitivity, specificity, positive predictive value, and negative predictive value of each system to correctly classify each drug pair as a DDI or not was calculated. Summary statistics of these measures were calculated separately for community and hospital pharmacies. RESULTS: Eight community pharmacies and 5 hospital pharmacies in the Tucson metropolitan area agreed to participate in the study. The median sensitivity and median specificity for community pharmacies was 0.88 (range, 0.81-0.94) and 0.91 (range, 0.67-1.00), respectively. For hospital pharmacies, the median sensitivity and median specificity was 0.38 (range, 0.15-0.94) and 0.95 (range, 0.81-0.95), respectively. CONCLUSIONS: Based on this convenience sample of 8 community pharmacies and 5 hospital pharmacies in 1 metropolitan area, the performance of community pharmacy computer systems in screening DDIs appears to have improved over the last several years compared with research published previously in 2001. However, significant variation remains in the performance of hospital pharmacy computer systems, even among systems manufactured by the same vendor. Future research should focus on improving the performance of these systems in accurately and precisely identifying DDIs with a high probability of resulting in true potential adverse effects on clinical outcomes and creating a low noise ratio associated with false-positive alerts.

METHODS: Ten community pharmacies and 10 hospital pharmacies in the Tucson metropolitan area were invited to participate in the study in 2004. To test the performance of each of the systems used by the pharmacies, 25 medications were used to create 6 mock patient profiles containing 37 drug-drug pairs, 16 of which are clinically meaningful DDIs that pose a potential risk to patient safety. Each profile was entered into the computer pharmacy system, and the system response in terms of the presence or absence of a DDI alert was recorded for each drug pair. The percentage of correct responses and the sensitivity, specificity, positive predictive value, and negative predictive value of each system to correctly classify each drug pair as a DDI or not was calculated. Summary statistics of these measures were calculated separately for community and hospital pharmacies.
RESULTS: Eight community pharmacies and 5 hospital pharmacies in the Tucson metropolitan area agreed to participate in the study. The median sensitivity and median specificity for community pharmacies was 0.88 (range, 0.81-0.94) and 0.91 (range, 0.67-1.00), respectively. For hospital pharmacies, the median sensitivity and median specificity was 0.38 (range, 0.15-0.94) and 0.95 (range, 0.81-0.95), respectively.
CONCLUSION: Based on this convenience sample of 8 community pharmacies and 5 hospital pharmacies in 1 metropolitan area, the performance of community pharmacy computer systems in screening DDIs appears to have improved over the last several years compared with research published previously in 2001. However, significant variation remains in the performance of hospital pharmacy computer systems, even among systems manufactured by the same vendor. Future research should focus on improving the performance of these systems in accurately and precisely identifying DDIs with a high probability of resulting in true potential adverse effects on clinical outcomes and creating a low "noise" ratio associated with false-positive alerts.. reviewed the performance of 9 community pharmacy systems in screening for 16 well-established DDIs and found up to one third of DDIs were not detected. 9 These 9 systems were used in more than 500 community pharmacies in Washington state, which suggested that thousands of patients could be exposed to potentially harmful DDIs that were missed by computerized DDI screening programs. To date, similar evaluations have not been performed on hospital pharmacy computer systems, and more recent evaluations of community pharmacy computer systems have not been conducted. This study was designed to update the findings of Hazlet et al. in community pharmacies and to provide additional data on hospital pharmacy systems. Thus, the purpose of this study was to evaluate the performance of DDI screening software installed on pharmacy computer systems in community and hospital pharmacies in identifying select clinically significant DDIs.

■■ Methods
A convenience sample of 10 community and 10 hospital pharmacies in the Tucson metropolitan area was recruited for this study in 2004. Each site was contacted by one of the investigators who explained the purpose of the study and invited them to participate. Sites were informed that neither the vendor name nor any pharmacy names would be reported in order to ensure the confidentiality of the participants. Because the purpose of the study was to evaluate the performance of different computer systems, only 1 pharmacy from each pharmacy chain was invited to participate. Ten community pharmacies representing 8 chain pharmacies and 2 independent pharmacies were invited to participate. Hospital pharmacies in the Tucson area were contacted prior to the evaluation of their pharmacy systems to ascertain the type of DDI software programs used by the hospitals; hospitals with the same pharmacy system were allowed. Ten hospital pharmacy departments were asked to participate.
The performance of each pharmacy computer system was evaluated using methodology similar to that published by Hazlet et al. 9 Twenty-five unique medications were used to create 6 mock individual patient profiles that incorporated different demographics and disease states (e.g., hypertension, coronary artery disease, depression, posttransplant). A total of 37 unique drug pairs were present in the patient profiles (Table 1). Of these, 16 were well-documented, clinically significant DDIs with the potential to cause harm to the patient. Two drug pairs were modified from the original list used by Hazlet et al. to account for the removal of cisapride (Propulsid) from the U.S. market. In the first profile, cisapride was replaced by ergotamine. In the second profile, cisapride was replaced by pimozide (Orap). These substitutions did not affect the number of drug interactions present in the profiles or the mechanism of each of the interactions (i.e., cytochrome P450 inhibition).
In addition, the clinical significance of the 16 DDIs was verified using 4 references (Drug Interaction Facts, 10 Dru g Interactions: Analysis and Management, 11 Evaluation of Dru g Interactions, 12 DRUG-REAX/Micromedex 13 ). Verification of clinical significance was based on the interaction listed as having (a) the highest severity rating (e.g., major, contraindicated, avoid combination) in at least 1 reference, or (b) a moderate severity rating in at least 2 of these references. These criteria were not clearly met by 1 drug pair: the erythromycin and sildenafil combination. The DRUG-REAX/Micromedex system rated the severity as moderate. Evaluation of Drug Interactions did not list the interaction. Drug Interaction Facts listed it as a having an overall significance rating of "4" on their 5-point scale, where "1" is the highest; however, the severity rating was listed as moderate. Drug Interactions: Analysis and Management assigned this interaction a significance rating of "3-minimize risk" on their 5-point rating scale. Because the severity ratings in the latter 2 references were somewhat equivocal in meeting the study criteria, the U.S. Food and Drug Administration (FDA)-approved prescribing information for sildenafil was used to verify the presence of the interaction.
Similarly, the absence of an interaction for the remaining drug pairs was evaluated by verifying the absence of the abovementioned criteria for each drug pair. Two drug pairs that were not considered DDIs met the criteria. The azithromycin and digoxin combination was listed as a "major" DDI by Evaluation of Drug Interactions and as a moderate interaction by DRUG-REAX/Micromedex. In both cases, the accompanying monographs referred to the interaction between erythromycin and digoxin and acknowledged the lack of information on azithromycin. The remaining references did not list the interaction; in addition, the prescribing information for azithromycin did not include an interaction with digoxin. Azithromycin and theophylline was listed as a moderate interaction by Drug Interaction Facts and DRUG-REAX/Micromedex. Drug Interactions: Analysis and Management specifically listed it as "No interaction"; Evaluation of Drug Interactions did not mention it. Again, both monographs referred to the interaction between erythromycin and theophylline and acknowledged that a pharmacokinetic study showed no significant effect of azithromycin on theophylline levels. In addition, the prescribing information for azithromycin states that no dosage adjustment is necessary when coadministering it with theophylline. Given that both these combinations were included in the Hazlet et al. 9 study as noninteractions and there were sufficient data to substantiate that claim, they were included as noninteractions in the present study as well.
One investigator visited each participating pharmacy to collect data on the performance of the pharmacy' s DDI software. Each patient profile was entered into the pharmacy system by the investigator or by a pharmacy staff member while the investigator observed. The pharmacy computer system' s response was assigned a severity level if either an alert was generated or there was no alert for each drug pair; it was recorded on a standard data collection form. The response was also printed whenever possible. Two exceptions to this process occurred in 1 community and in 1 hospital pharmacy. One community pharmacy did not allow the investigator to be on-site during testing so it was performed by the pharmacy manager and the results were sent to the investigators. In the case of the hospital pharmacy exception, the investigators obtained a comprehensive list of all the DDIs and corresponding severity ratings that are used by the hospital pharmacy system and assessed the performance using this information.
The percentage correct was calculated for responses from community and hospital pharmacies for each individual drug pair and overall. In addition, an overall percentage correct was calculated for corresponding true-positive DDIs between this study and those reported by Hazlet et al. 9 ; i.e., this included only true-positive DDIs that were used in both studies. Performance was assessed by calculating the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of DDI alerting for each of the systems. 14 Sensitivity is the probability of a DDI alert given that DDI is present: sensitivity = number of true-positive DDI alerts/number of DDI pairs. Specificity is the probability of the absence of an alert given that a DDI is not present: specificity = number of truenegative nonalerts/number of non-DDI pairs. Positive predictive value is a measure of the usefulness of the alert in that it is the probability that a DDI alert represents a true DDI: PPV = (number of true positives)/(number of true positives + number of false positives). Conversely, negative predictive value is the probability that the absence of a DDI alert represents a true absence of a DDI: NPV = (number of true negatives)/(number of true negatives + number of false negatives). These measures were also summarized by calculating descriptive statistics for the community and hospital pharmacy programs that were included in this study. Performance measures for community pharmacies published by Hazlet et al. 9 were included for comparison. This study was reviewed and approved by the University of Arizona Human Subjects Protection Committee.

■■ Results Community Pharmacy Systems
Eight community pharmacies agreed to participate in this study. Of the 8, 3 were using different versions of computer systems from the same vendor. The overall median sensitivity of the software systems in screening the 16 DDI pairs was 0.88 (range, 0.81-0.94). The overall median specificity of the systems was 0.91 (range, 0.67-1.00). The overall median PPV score was 0.88 (range, 0.68-1.00). The overall median NPV was 0.91 (range, 0.86-0.95). Twelve of the 16 DDI pairs were correctly classified by all the computer systems. Table 2 shows the results for each system.

Hospital Pharmacy Systems
Five hospital pharmacies agreed to participate in the study. Three of the 5 were using the same pharmacy computer system. It is important to note that the computer systems were individualized in that interactions could be turned on or off at the discretion of the pharmacy staff or manager. The overall median sensitivity of the software systems was 0.38 (range, 0.15-0.94). The overall median specificity was 0.95 (range, 0.81-0.95). The overall median PPV and NPV were 0.83 (range, 0.67-0.94) and 0.67 (range, 0.64-0.95), respectively. Table 3 shows the results by individual hospital. It was not possible to enter phenelzine in the computer at the third acute care institution. Thus, the measures for this institution were calculated using only 34 drug pairs. Taking into account the inability to include phenelzine by considering it a false negative would have lowered the sensitivity and NPV of this system to 0.13 and 0.59, respectively (specificity and PPV would remain unchanged).

Individual Drug-Drug Pairs
Of the 37 drug-drug pairs included in this study, 11 were correctly identified by all community and hospital pharmacy systems. Ten of the 11 were not DDIs. Ketoconazole -simvastatin was correctly identified as a DDI by all systems. One drug pair   considered to be a true-positive DDI, atenolol -digoxin, was incorrectly identified (missed) by all of the pharmacy systems evaluated in this study.

■■ Discussion
The findings from the present study indicate that many pharmacy computer systems may be operating at low levels of sensitivity and specificity when screening for DDIs. Among community pharmacy systems, just over 1 in 11 clinically significant DDIs were not detected. Of the 8 computer systems that were included in this study, all performed with a sensitivity of at least 0.81, meaning that, in the worst-case scenario, they would miss approximately 1 out of every 5 clinically important DDIs. Among hospital pharmacy computer systems, the median sensitivity was 0.38, which indicates that, for the hospitals included in this study, the majority of DDIs presented would be missed. Some systems performed at levels where they would only detect approximately 1 out of every 7 DDIs that were encountered.
This study mirrored previous work done by Hazlet et al. The reported median sensitivity and specificity of the community pharmacy systems in that study were 0.71 (range, 0.44-0.88) and 0.89 (range, 0.71-1.00), respectively. 9 The results of this study indicate that improvement has occurred in the sensitivity, PPV, and NPV among community pharmacy systems in detecting the 16 DDIs. The specificity of the systems in this study was comparable to that reported by Hazlet. Several reasons could explain these findings, including improvements in the quality and quantity of published literature supporting the existence of or lack of significance of a particular DDI and improvements in the methods used by software vendors to incorporate DDI information into their software products. The fact that minor changes were made in the drug combinations used in this study and the evaluation of potentially different pharmacy computer systems could also explain the difference. More recently, Barrons analyzed the performance of 9 personal digital assistant software programs in evaluating 80 drug pairs, 40 of which were DDI pairs developed by using eFacts, Micromedex DRU-GREAX, and Drug Interactions: Analysis and Management. The results of this study were comparable to the results from the community pharmacies in the present study in that the median sensitivity and specificity for these software programs was 0.95 (range, 0.87-1.0) and 0.85 (range, 0.70-1.0), respectively; the median PPV and median NPV was 0.95. 15 The correct identification of the selected DDIs by community pharmacy systems included in this study improved compared with the results presented by Hazlet et al. 9 Of 13 specific DDIs that were included in both studies, 10 were correctly classified by all the community pharmacy systems; only one of these interactions was correctly classified by all the pharmacy systems evaluated by Hazlet et al. One DDI, atenolol -digoxin, was incorrectly classified (i.e., missed as a true-positive interaction) by all pharmacy systems, which caused a decrease in sensitivity compared with the previous study. 9 However, this drug combination is frequently used together for rate control in atrial fibrillation, a common disease state. Many patients take this drug combination of atenolol and digoxin safely with appropriate monitoring, which may have led developers of these computer system DDI modules to exclude this particular DDI to avoid the nuisance of frequent overrides for false-positive alerts, a common problem in DDI alerting. Among hospital pharmacy systems, 10 DDIs were detected by less than 50% of systems in this study. Of the 15 medications involved in these 10 DDIs, the most frequently involved were tacrolimus, fluconazole, erythromycin, and verapamil.
Several factors can contribute to the performance of systems that screen for DDIs with a lower sensitivity. The results of this study suggest that a potential factor within hospital systems may be the customization of pharmacy computer systems, particularly the DDI alerts. Enabling the customization of computer systems by users (i.e., individuals or institutions) is an essential part of creating a product that is user-friendly and functional in any setting. However, it may also contribute to variation in the performance of the system to the point where patient care may be compromised. Three of the 5 hospitals in this study used hospital pharmacy systems manufactured by the same vendor. Because the hospitals and computer systems were not linked, in order to protect confidentiality, it was not possible to identify the hospitals using the same vendor. Yet, the 3 systems with the most similar performance still had sensitivity values with a range of 0.19 (range, 0.31-0.5). It is also possible that the sensitivity values for these 3 same-vendor systems had a range as large as 0.79 (range, 0.15-0.94). Previous research has implicated customization as a potential factor contributing to missed DDIs among community pharmacy computer systems. 9 Whether or not the missed DDIs in this study were a result of customization or complete absence from the system was not explored in this study.
Previous studies have found discrepancies in DDI references and computerized screening systems. 16 Whether or not the missed DDIs in this study were a result of customization or complete absence from the system was not explored in this study.
A challenge in assessing the performance of pharmacy computer systems in detecting DDIs is the absence of benchmarks with which to compare against and judge the appropriateness of the results. Broad goals have been set with respect to improving patient safety and reducing the number of adverse drug events related to medication errors. 1 Certainly, reducing exposures to potentially harmful DDIs is a step forward in this effort. However, little is known about the optimal level of DDI alerting that should be present in order to achieve this goal. 4 It would seem ideal to have DDI alerting systems operate at 100% sensitivity and specificity for all DDIs. Yet, this is likely to create a situation where the signal-to-noise ratio for clinically important DDIs is low, and operating a system with this level of sensitivity and specificity is unrealistic. Most systems operate at a level below 100%, as shown in this study. The question that needs to be answered in order to determine an appropriate benchmark is how low can the level of sensitivity and specificity be without increasing the rate of clinically important adverse events resulting from DDIs?
Determining an appropriate benchmark is complicated by the fact that DDIs vary in terms of their severity, predictability, and incidence, and not enough is known about these factors for most DDIs. Previous studies have found discrepancies in DDI references and computerized screening systems with respect to classifying the severity of DDIs. 5,8 More recently, Abarca et al. found practically no agreement between 4 commonly used DDI references with regard to the classification of the severity of interactions considered to be of highest clinical importance. 7 Armstrong and Denemark evaluated pharmacists' responses to drug utilization review (DUR) alerts, which included DDI alerts, in a Medicaid population. 17 Over a 1-year period, 0.81% of prescription claims generated a DDI alert. Yet, only 6.1% of these alerts resulted in a medication not being dispensed. More recently, Peng et al. evaluated the incidence of 69 DDI pairs in a drug claims database over a 1-year period and found a DDI incidence of 0.8% using simple screening. Using a retrospective, computerized DUR program with sophisticated DDI filters followed by a clinical pharmacist audit, they reduced the number of false-positive alerts by 94.3%, yielding an incidence of 0.04%. 18 While the overall incidence of DDIs has been shown to be relatively low, the incidence of individual DDIs can vary dramatically. 19 The relatively rare occurrence of DDIs and the associated variance contributes to the difficulty in designing alerting systems that detect with a high sensitivity and specificity across a large number of DDIs. In many respects, the issues facing DDI alert systems also are problems for other computer features aimed at preventing medical errors. 4 With an aging population, the number of patients on multiple medications will increase and the likelihood of clinically significant DDIs will escalate. 6,20 The need for more specific DDI screening tools increases as the number of prescriptions being filled rises. 20

Limitations
It is important to note several limitations of the present study. First, a relatively small number of drug pairs and DDIs were included in this evaluation, and there are many more drug interactions that are considered to be of high clinical importance that were not examined in the present study. A larger number of test DDIs would have been desirable, but this had to be balanced against the amount of work required to test them, the desire to limit the burden on participating pharmacies, and the desire to be able to compare the results with that of a previous study. Second, the results of the present study are based on a small sample of community and hospital pharmacies in 1 metropolitan area, which may limit the generalizability of these findings to other settings. However, the computer systems in use at the sites included in the present study are widely used and increase the applicability of these results to areas outside of this metropolitan area. The pharmacy systems used by the 8 community pharmacies included in this study are also used at more than 27,500 community pharmacies across the United States. Similarly, the pharmacy systems evaluated among the participating hospitals are used at a minimum of 1,500 hospitals in the United States (exact numbers were not available for all vendors so this estimate is conservative). While it is likely that customization of the hospital systems at the local level contributed to the variation in sensitivity, the salient features of the drug interaction modules are still represented and customization is likely to affect pharmacies not included in the sample in a similar manner.
Third, the present study did not attempt to link the performance of the pharmacy system with actual patient outcomes. The clinical significance of the performance of these computer systems in identifying and classifying DDIs is therefore not known. Any reference to an improvement or quality shortfall in the performance of any pharmacy system should not be interpreted as a change in actual patient outcomes.

■■ Conclusion
The performance of community pharmacy computer systems in screening the selected DDIs appears to have improved over the last several years. However, there appears to be significant variation in the performance of hospital pharmacy computer systems in the screening of DDIs, even among systems manufactured by the same vendor. Quality improvement efforts should focus on improving the performance of these systems in flagging DDIs with a high probability of true-positive adverse clinical effects and ignoring DDIs that have a high probability of having no adverse clinical effects.